Skip to main content

Web Crawls

The Web Archive of the Internet Archive started in late 1996, is made available through the Wayback Machine, and some collections are available in bulk to researchers. Many pages are archived by the Internet Archive for other contributors including partners of Archive-IT, and Save Page Now users. Other captures are donated to the Internet Archive by other partners such as Alexa Internet.

3,801,586
RESULTS
rss


Media Type
8,212
collections
3,682,619
web
87,077
movies
22,450
data
480
audio
374
texts
More right-solid
Topics & Subjects
1,244,531
crawldata
976,223
wiki
949,663
dumps
896,894
incremental
362,121
Wikipedia
188,569
Wiktionary
More right-solid
Collection
More right-solid
Creator
1,037,705
internet archive
902,391
wikimedia projects editors
285,723
archive-it
116,099
thumper2.php
73,846
internetarchive
33,026
alexa internet
More right-solid
Language
138,600
English
11,042
German
9,657
Dutch
9,369
Portuguese
9,230
Hungarian
9,174
Korean
More right-solid
SHOW DETAILS
up-solid down-solid
eye
Title
Date Archived
Creator
Internet Archive Web Crawls
collection
1,026,277
ITEMS
8.7B
VIEWS
collection
eye 8.7B
The Internet Archive discovers and captures web pages through many different web crawls. At any given time several distinct crawls are running, some for months, and some every day or longer. View the web archive through the Wayback Machine .
Topic: webwidecrawl
Worldwide Web Crawls
collection
519,296
ITEMS
3.7B
VIEWS
collection
eye 3.7B
Wide crawls of the Internet conducted by Internet Archive. Please visit the Wayback Machine to explore archived web sites. Since September 10th, 2010, the Internet Archive has been running Worldwide Web Crawls of the global web, capturing web elements, pages, sites and parts of sites. Each Worldwide Web Crawl was initiated from one or more lists of URLs that are known as "Seed Lists". Descriptions of the Seed Lists associated with each crawl may be provided as part of the metadata for...
Alexa Crawls
collection
161,237
ITEMS
3.6B
VIEWS
collection
eye 3.6B
Starting in 1996, Alexa Internet has been donating their crawl data to the Internet Archive. Flowing in every day, these data are added to the Wayback Machine after an embargo period.
Topics: web crawl, Alexa
Survey Crawls
collection
74,135
ITEMS
2B
VIEWS
collection
eye 2B
Survey crawls are run about twice a year, on average, and attempt to capture the content of the front page of every web host ever seen by the Internet Archive since 1996.
Topic: survey crawls
Live Web Proxy Crawls
collection
22,380
ITEMS
1.7B
VIEWS
collection
eye 1.7B
Content crawled via the Wayback Machine Live Proxy mostly by the Save Page Now feature on web.archive.org. Liveweb proxy is a component of Internet Archive’s wayback machine project. The liveweb proxy captures the content of a web page in real time, archives it into a ARC or WARC file and returns the ARC/WARC record back to the wayback machine to process. The recorded ARC/WARC file becomes part of the wayback machine in due course of time.
Archive-It Digital Collection
collection
296,098
ITEMS
945.2M
VIEWS
collection
eye 945.2M
Archive-It is a subscription web archiving service of the Internet Archive that helps organizations harvest, build, and preserve collections of digital content. Partners create domain specific collections of web captures that can be searched on Archive It . Content is hosted and stored at the Internet Archive data centers. Archive-It works with more than 400 partner organizations in 48 U.S. states and 16 countries worldwide including: College and University Libraries State Archives, Libraries,...
Topic: Colleges, Universities, Libraries, Archives, NGOs, Museums
Archive-It Partners
collection
291,564
ITEMS
932.5M
VIEWS
collection
eye 932.5M
Archive-It is the leading web archiving service for collecting and accessing cultural heritage on the web and is a service of Internet Archive used by libraries, archives, governments, non-profits, and other organizations to build collections of web materials.
Topic: TK
Focused Crawls
collection
198,464
ITEMS
726.1M
VIEWS
by Internet Archive
collection
eye 726.1M
Focused crawls are collections of frequently-updated webcrawl data from narrow (as opposed to broad or wide) web crawls, often focused on a single domain or subdomain.
Topic: webcrawl
collection
eye 662.3M
The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
Custom Crawl Services
collection
62,105
ITEMS
562.6M
VIEWS
by Internet Archive
collection
eye 562.6M
Large-scale web harvests and national domain crawls performed for National Libraries, National Archives, preservation partners, research initiatives, and as part of special projects and custom crawling and research services.
Topic: ccs
Fix Broken Links Web Crawls
collection
68,893
ITEMS
532.3M
VIEWS
collection
eye 532.3M
These crawls are part of an effort to archive pages as they are created and archive the pages that they refer to. That way, as the pages that are referenced are changed or taken from the web, a link to the version that was live when the page was written will be preserved. Then the Internet Archive hopes that references to these archived pages will be put in place of a link that would be otherwise be broken, or a companion link to allow people to see what was originally intended by a page's...
web-group-internal
collection
31,905
ITEMS
513.4M
VIEWS
collection
eye 513.4M
miscellaneous data
Topic: brad tofel
Top Domains
collection
99,635
ITEMS
498M
VIEWS
collection
eye 498M
A daily collection of thousands of the most popular web sites according to Alexa.com's top sites rankings .
Topics: daily, popular sites, Alexa
Wide Crawl started April 2013
collection
25,005
ITEMS
481.1M
VIEWS
collection
eye 481.1M
Web wide crawl with initial seedlist and crawler configuration from April 2013.
collection
eye 420.5M
The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
Wayback Indexes
collection
554
ITEMS
412.6M
VIEWS
collection
eye 412.6M
Wayback indexes. This data is currently not publicly accessible.
alexa_2007
collection
7,636
ITEMS
408.3M
VIEWS
collection
eye 408.3M
this data is currently not publicly accessible.
Archive Team
collection
269,131
ITEMS
394.9M
VIEWS
collection
eye 394.9M
Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history. History is littered with hundreds of conflicts over the future of a community, group, location or...
Wiki Collections
collection
1,343,023
ITEMS
371.6M
VIEWS
collection
eye 371.6M
Collections of Wiki data
Topics: crawls, data, wiki
Wikipedia Outlinks
collection
28,874
ITEMS
366.7M
VIEWS
collection
eye 366.7M
Crawl of outlinks from wikipedia.org . These files are currently not publicly accessible. from Wikipedia : Wikipedia is a multilingual, web-based, free-content encyclopedia project operated by the Wikimedia Foundation and based on an openly editable model. The name "Wikipedia" is a portmanteau of the words wiki (a technology for creating collaborative websites, from the Hawaiian word wiki, meaning "quick") and encyclopedia. Wikipedia's articles provide links to guide the...
Wide Crawl started June 2014
collection
45,313
ITEMS
349.4M
VIEWS
collection
eye 349.4M
Web wide crawl with initial seedlist and crawler configuration from June 2014.
collection
eye 340.7M
The seed for Wide00014 was: - Slash pages from every domain on the web: -- a list of domains using Survey crawl seeds -- a list of domains using Wide00012 web graph -- a list of domains using Wide00013 web graph - Top ranked pages (up to a max of 100) from every linked-to domain using the Wide00012 inter-domain navigational link graph -- a ranking of all URLs that have more than one incoming inter-domain link (rank was determined by number of incoming links using Wide00012 inter domain links)...
Wide Crawl Number 12 - started March, 14th 2015
collection
49,621
ITEMS
320.5M
VIEWS
collection
eye 320.5M
Web wide crawl with initial seedlist and crawler configuration from January 2015.
Wide Crawl started August 2013
collection
21,911
ITEMS
304.2M
VIEWS
collection
eye 304.2M
Web wide crawl with initial seedlist and crawler configuration from August 2013.
Wide Crawl started January 2012
collection
30,362
ITEMS
290.2M
VIEWS
collection
eye 290.2M
Web wide crawl with initial seedlist and crawler configuration from January 2012 using HQ software.
alexa_2006
collection
6,507
ITEMS
288.2M
VIEWS
collection
eye 288.2M
this data is currently not publicly accessible.
Wikipedia Near Real Time (from IRC)
collection
16,010
ITEMS
274.4M
VIEWS
collection
eye 274.4M
This is a collection of web page captures from links added to, or changed on, Wikipedia pages. The idea is to bring a reliability to Wikipedia outlinks so that if the pages referenced by Wikipedia articles are changed, or go away, a reader can permanently find what was originally referred to. This is part of the Internet Archive's attempt to rid the web of broken links .
Topics: Wikipedia, Wikimedia
collection
eye 268M
The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
Wide Crawl started April 2012
collection
39,252
ITEMS
259.1M
VIEWS
collection
eye 259.1M
Web wide crawl with initial seedlist and crawler configuration from April 2012.
collection
eye 241.3M
The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
collection
eye 212.8M
The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
ArchiveBot: The Archive Team Crowdsourced Crawler
collection
14,178
ITEMS
209.5M
VIEWS
collection
eye 209.5M
ArchiveBot is an IRC bot designed to automate the archival of smaller websites (e.g. up to a few hundred thousand URLs). You give it a URL to start at, and it grabs all content under that URL, records it in a WARC, and then uploads that WARC to ArchiveTeam servers for eventual injection into the Internet Archive (or other archive sites). To use ArchiveBot, drop by #archivebot on EFNet. To interact with ArchiveBot, you issue commands by typing it into the channel. Note you will need channel...
Topics: archiveteam, archivebot, webcrawl, robot, love
Wide Crawl Number 13
collection
46,049
ITEMS
198.1M
VIEWS
collection
eye 198.1M
Web Wide Crawl Number 13
Wide Crawl started October 2010
collection
15,839
ITEMS
190M
VIEWS
collection
eye 190M
Web wide crawl with initial seedlist and crawler configuration from October 2010
Wide Crawl Started January 2013
collection
15,138
ITEMS
187.9M
VIEWS
collection
eye 187.9M
Wide crawls of the Internet conducted by Internet Archive. Access to content is restricted. Please visit the Wayback Machine to explore archived web sites.
Wide Crawl started September 2012
collection
22,402
ITEMS
186.7M
VIEWS
collection
eye 186.7M
Web wide crawl with initial seedlist and crawler configuration from September 2012.
Top News
collection
78,233
ITEMS
175.7M
VIEWS
collection
eye 175.7M
A daily collection of hundreds of the world's top news sites.
Topics: daily, news
Wide Crawl started October 2011
collection
12,648
ITEMS
166.5M
VIEWS
collection
eye 166.5M
Web wide crawl with initial seedlist and crawler configuration from March 2011 using HQ software.
Around The World Crawl
collection
2,150
ITEMS
160.5M
VIEWS
collection
eye 160.5M
Data crawled by Sloan Foundation on behalf of Internet Archive
Wikipedia Outlinks March 2016
collection
23,270
ITEMS
155.7M
VIEWS
collection
eye 155.7M
Crawl of outlinks from wikipedia.org started March, 2016. These files are currently not publicly accessible. Properties of this collection. It has been several years since the last time we did this. For this collection, several things were done: 1. Turned off duplicate detection. This collection will be complete, as there is a good chance we will share the data, and sharing data with pointers to random other collections, is a complex problem. 2. For the first time, did all the different wikis....
National Library of Australia Crawls
collection
21,982
ITEMS
154.7M
VIEWS
collection
eye 154.7M
Crawls performed by Internet Archive on behalf of the National Library of Australia. This data is currently not publicly accessible.
Wide Crawl started February 2014
collection
9,789
ITEMS
153.4M
VIEWS
collection
eye 153.4M
Web wide crawl with initial seedlist and crawler configuration from February 2014.
.com survey started January 2011
collection
2,535
ITEMS
149.2M
VIEWS
collection
eye 149.2M
Survey crawl of .com domains started January 2011.
Topic: webcrawl
Wide Crawl started March 2011
collection
8,528
ITEMS
148.3M
VIEWS
collection
eye 148.3M
Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi. What’s in the data set: Crawl start date: 09 March, 2011 Crawl end date: 23 December, 2011 Number of captures: 2,713,676,341 Number of unique URLs: 2,273,840,159 Number of hosts: 29,032,069 The seed list for this crawl was a list of Alexa’s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT)...
Wordpress Blogs and the Pages They Link To
collection
18,121
ITEMS
139.3M
VIEWS
collection
eye 139.3M
This is a collection of pages and embedded objects from WordPress blogs and the external pages they link to. Captures of these pages are made on a continuous basis seeded from a feed of new or changed pages hosted by Wordpress.com or by Wordpress pages hosted by sites running a properly configured Jetpack wordpress plugin.
Topics: Wordpress.com, blogs, jetpack
alexa_web_2009
collection
3,080
ITEMS
138.7M
VIEWS
collection
eye 138.7M
this data is currently not publicly accessible.
alexa_web_2010
collection
2,994
ITEMS
132.8M
VIEWS
collection
eye 132.8M
this data is currently not publicly accessible.
38_crawl
collection
1,387
ITEMS
127M
VIEWS
collection
eye 127M
this data is currently not publicly accessible.
Wikipedia Outlinks February 2012
collection
2,951
ITEMS
118.5M
VIEWS
collection
eye 118.5M
Crawl of outlinks from wikipedia.org started February, 2012. These files are currently not publicly accessible.
web_wk
collection
9,978
ITEMS
107.1M
VIEWS
collection
eye 107.1M
Crawl performed by Internet Archive. This data is currently not publicly accessible.
Alexa Crawl EG
collection
1,678
ITEMS
106.9M
VIEWS
collection
eye 106.9M
Crawl EG from Alexa Internet. This data is currently not publicly accessible.
web_iq
collection
2,650
ITEMS
101.5M
VIEWS
collection
eye 101.5M
Crawl performed by Internet Archive. This data is currently not publicly accessible.
National Library of Spain Crawls
collection
6,741
ITEMS
101.4M
VIEWS
collection
eye 101.4M
Data collected by Internet Archive on behalf of the National Library of Spain. This data is currently not publicly accessible.
collection
eye 96.3M
Web wide crawl.
GDELT
collection
33,367
ITEMS
90.2M
VIEWS
collection
eye 90.2M
A daily crawl of more than 200,000 home pages of news sites, including the pages linked from those home pages. Site list provided by The GDELT Project
Topics: GDELT, News
51_crawl
collection
1,138
ITEMS
86.6M
VIEWS
collection
eye 86.6M
this data is currently not publicly accessible.
52_crawl
collection
2,589
ITEMS
86.5M
VIEWS
collection
eye 86.5M
this data is currently not publicly accessible.
26_crawl
collection
1,466
ITEMS
84M
VIEWS
collection
eye 84M
this data is currently not publicly accessible.
Archive Team: The News Roundup
collection
29,322
ITEMS
75M
VIEWS
by Archive Team
collection
eye 75M
Archive Team now searches many, many news sites, including extensive worldwide and obscure sources, to capture unique news stories for history.
Bibliotheque Nationale de France Domain Crawls
collection
1,652
ITEMS
70.4M
VIEWS
collection
eye 70.4M
Crawls of the french domain space performed by Internet Archive on behalf of Bibliotheque Nationale de France. This data is currently not publicly accessible.
35_crawl
collection
1,179
ITEMS
65.9M
VIEWS
collection
eye 65.9M
this data is currently not publicly accessible.
Shallow Crawls
collection
1,042
ITEMS
61.4M
VIEWS
collection
eye 61.4M
Shallow crawls that collect content 1 level deep including embeds. This data is currently not publicly accessible.
Alexa Crawl EI
collection
1,408
ITEMS
61.2M
VIEWS
collection
eye 61.2M
Crawl EI from Alexa Internet. This data is currently not publicly accessible.
International News Crawls
collection
4,422
ITEMS
61.2M
VIEWS
collection
eye 61.2M
Crawls of International News Sites
29_crawl
collection
1,568
ITEMS
58.4M
VIEWS
collection
eye 58.4M
this data is currently not publicly accessible.
Internet Archive Global Events
collection
7,712
ITEMS
57.7M
VIEWS
collection
eye 57.7M
Internet Archive Global Events Archive-It Partner Since: Feb, 2006 Organization Type: Other Institutions Organization URL: http://www.archive-it.org
Alexa Crawl DX
collection
1,442
ITEMS
55.7M
VIEWS
collection
eye 55.7M
Crawl DX from Alexa Internet. This data is currently not publicly accessible.
Alexa Crawls DF
collection
248
ITEMS
54.8M
VIEWS
collection
eye 54.8M
Crawl data donated by Alexa Internet. This data is currently not publicly accessible
web_mon
collection
3,810
ITEMS
54.6M
VIEWS
collection
eye 54.6M
Crawl performed by Internet Archive. This data is currently not publicly accessible.
Wikipedia Outlinks May 2011
collection
1,638
ITEMS
54.4M
VIEWS
collection
eye 54.4M
Crawl of outlinks from wikipedia.org started May, 2011. These files are currently not publicly accessible.
web_el_2008
collection
1,705
ITEMS
53.5M
VIEWS
collection
eye 53.5M
This data is currently not publicly accessible.
alexa_1999
collection
243
ITEMS
53.2M
VIEWS
collection
eye 53.2M
this data is currently not publicly accessible.
Elections Web
collection
1,609
ITEMS
52.7M
VIEWS
collection
eye 52.7M
This collection contains collaborative Election crawls performed by IA.
Topics: elections, web
Election Crawl 2012
collection
1,608
ITEMS
52.7M
VIEWS
collection
eye 52.7M
This crawl was performed in Summer & Fall of 2012 to archive the US Federal Elections.
Topics: US, federal, elections, web, 2012
Alexa Crawls EA
collection
1,315
ITEMS
51.5M
VIEWS
collection
eye 51.5M
Crawl data donated by Alexa Internet. This data is currently not publicly accessible
Topic: crawldata